Project : Discerning Bots from Humans in Digital Spaces: An Advanced AI Detection Approach

Leveraging Explainable AI for Enhanced Interpretability

Led By: Heon Kim, Vincent Caldwell, Mike Meissner

</div>

Table of Contents

  1. Importing packages
  2. Helper functions
  3. Importing data
  4. Feature engineering
  5. Cleaning tweets
  6. Sentiment scores using NLTK and VADER lexicon
  7. Additional Feature Engineering
  8. Exploratory Data Analaysis
  9. Machine Learning Models
  10. Neural Networks/Deep Learning
  11. Enhancing models with feature selections
  12. Random Forest Classification Models
  13. Using LIME (explainable AI)
  14. SHAP (explainable AI)

Importing packages

Back to Top

Helper functions

Back to Top

Importing data

Summary Statistics for Numerical Columns

The numerical columns provide insights into various aspects of the dataset. Here are some key observations:

Metrics related to tweets' content and user activity: Columns like tweet_length, word_count, readability_score, and sentiment show a broad distribution, which could indicate diverse content and writing styles among the users.

Overview of Categorical Data:

Back to Top

Feature engineering

Cleaning tweets

Back to Top

Sentiment scores using NLTK and VADER lexicon

This code snippet is performing cluster analysis on a DataFrame (df) using only one feature, follower_count. It aims to identify the optimal number of clusters that best represents the underlying structure of the data. The process involves applying the K-Means clustering algorithm with a range of cluster numbers and evaluating the results using two common metrics: the Elbow Method and the Silhouette Score. Let's break down what's happening step by step:

Preparation for Clustering

X = df[['follower_count']]: This line creates a new DataFrame, X, which contains only the follower_count column from the original DataFrame df. This is the feature based on which the clustering will be performed.

Variables to Store Results

elbow_scores = []: An empty list to store the inertia scores from the K-Means algorithm for different numbers of clusters. Inertia measures how internally coherent the clusters are.

silhouette_scores = []: An empty list to store the silhouette scores for different numbers of clusters. The silhouette score is a measure of how similar an object is to its own cluster compared to other clusters.

range_of_clusters = range(2, 11): Defines the range of cluster numbers to be tested, from 2 to 10. The choice of starting with 2 is because a single cluster is trivial and provides no meaningful segmentation.

Loop Over the Range of Cluster Numbers

The loop iterates over each number of clusters specified in range_of_clusters. kmeans = KMeans(n_clusters=n_clusters, random_state=42): Initializes the K-Means algorithm with the current number of clusters (n_clusters) and a fixed random_state for reproducibility.

cluster_labels = kmeans.fit_predict(X): Fits the K-Means algorithm to the data X and assigns each data point to a cluster. The cluster labels are stored in cluster_labels.

elbowscores.append(kmeans.inertia): Calculates and stores the inertia (Elbow Method score) of the current model. Inertia is a measure of the sum of squared distances of samples to their nearest cluster center.

silhouette_avg = silhouette_score(X, cluster_labels): Calculates the average silhouette score for the current clustering. The silhouette score ranges from -1 to 1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.

silhouette_scores.append(silhouette_avg): Stores the average silhouette score.

Plotting the Results

The code then plots two graphs in a single figure with two subplots:

Elbow Method Plot: Shows the inertia (sum of squared distances within clusters) for each number of clusters. A lower inertia indicates tighter clusters. The "elbow" point, where the rate of decrease sharply changes, suggests an optimal number of clusters.

Silhouette Score Plot: Shows the silhouette score for each number of clusters. A higher silhouette score indicates better defined clusters. The optimal number of clusters is often where the silhouette score is highest.

Interpretation

By analyzing these plots, one can determine the optimal number of clusters to use for further analysis. The Elbow Method helps identify a point after which the inertia decreases at a slower rate, suggesting diminishing returns by adding more clusters. The Silhouette Score evaluates how well each object lies within its cluster, with higher scores indicating better clustering.

This approach, combining the Elbow Method and Silhouette Score, provides a comprehensive way to assess the clustering quality and decide on the most appropriate number of clusters for the dataset based on follower_count.

Back to Top

Clustering

Back to Top

Additional Feature Engineering

Back to Top

Wordcloud of the 50 most common words for bots and nots

Back to Top

Indicate if the tweet contains the top 50 words from bot or not

Back to Top

Exploratory Data Analaysis

The heatmap provides a visualization of correlation coefficients between various social media account features and their likelihood of being bots, with a special focus on the 'bot_label' as a reference for bot classification. Key takeaways include most features showing negligible correlation with bot detection, indicating a lack of strong linear relationships to determine bot status. Specifically, features like 'contains_unique_botwords' and 'contains_unique_nonbotwords' exhibit weak positive and negative correlations, respectively, but fail to serve as strong predictors for bot identification. This suggests the need for more complex models or additional feature engineering to enhance predictive accuracy. The heatmap highlights the necessity for further analysis, potentially involving more sophisticated statistical methods or machine learning algorithms, to uncover deeper patterns or feature interactions not immediately apparent from initial observations.

The bar chart compares sentiment distribution—positive, neutral, and negative—among verified and unverified social media accounts, revealing a predominant presence of positive sentiments across both account types. Notably, verified accounts display slightly more positive sentiments, while unverified accounts lean marginally more towards neutral sentiments; the counts for negative sentiments are closely matched between the two. Despite these differences, the overall sentiment ratio remains consistent, suggesting similar sentiment expression patterns regardless of verification status. This consistency implies that verification may not significantly impact the sentiment of the content posted, though the observed subtleties suggest potential areas for further investigation. The chart suggests that the dynamics of sentiment expression on social media do not drastically differ between verified and unverified users, indicating a broad appeal of the platform that transcends verification status. For more definitive insights, additional statistical analysis could explore whether the observed differences are meaningful or merely by chance, enhancing the understanding of sentiment distribution dynamics across different user types on the platform.

The time series graph depicts the weekly average sentiment scores from January 2020 to January 2023, showcasing fluctuations that mirror changing public moods or reactions to events, without indicating a clear long-term trend. The sentiment scores, oscillating mildly positive between 0.12 and 0.22, suggest a generally optimistic sentiment baseline. Potential seasonal trends or the impact of specific events on sentiment could be discerned with more detailed analysis. The graph hints at outliers possibly tied to significant events, emphasizing the importance of contextual analysis to understand sentiment shifts. Additionally, the consistency in data collection and analysis methodologies is crucial for accurate sentiment tracking over time. This graph highlights the dynamic nature of public sentiment, underlining the need for further investigation, including external data, to fully grasp the underlying factors influencing these sentiment trends.

Back to Top

Geographical and Demographical Insights

Geographical and Demographical Insights Analysis

Distribution of Bots by Location (Top 10 Locations)

Verified Accounts vs. Bot Label

Back to Top

Temporal Analysis

Temporal Analysis Results

Tweet Activity Patterns Over Time

Account Age Analysis

The distribution of account ages shows that both bots and human users span a range of account ages, with some peaks and troughs. The Kernel Density Estimate (KDE) overlay indicates the distribution's shape, showing that both newer and older accounts are represented among bots and humans. It's noteworthy to examine if certain age ranges are more prevalent among bots, which could suggest strategies in bot creation and deployment over time.

Insights and Recommendations

Back to Top

Engagement and Content Analysis

Engagement Metrics Correlation

The correlation matrix for engagement metrics (retweet_count, mention_count, follower_count) reveals very low correlation coefficients among these variables. This indicates that, at least linearly, there's little to no direct relationship between these metrics within the dataset. For instance, the number of retweets doesn't strongly correlate with the number of mentions or the follower count, suggesting that high engagement in one aspect doesn't necessarily mean high engagement in others.

Sentiment Analysis

Advanced Bot Behavior Analysis

Given the initial findings from the engagement and sentiment analysis, it's evident that understanding bot behavior requires a multifaceted approach, considering not just one or two features, but a combination of them.

Back to Top

The stacked bar chart illustrates monthly tweet activity, distinguishing between human and bot contributions. It shows consistent engagement from both groups over time, with bots sometimes matching or exceeding human tweet volumes, highlighting their significant presence. Despite this, humans also maintain considerable activity levels, indicating the platform's role in fostering human interaction. The tweet volume ratio between humans and bots remains stable across the observed period, suggesting unchanged account behaviors or platform policies. Further analysis, especially of tweet content, could provide insights into bots' roles in information dissemination or influence. The accuracy of distinguishing bots from humans is critical for the validity of these observations, underscoring the importance of the classification methodology. Overall, the chart effectively portrays the dynamic interaction between humans and bots on the platform.

Back to Top

Language Complexity and Diversity

The boxplots compare 'Lexical Diversity' and 'Readability Score' between human and bot accounts on social media. Both groups show similar lexical diversity, with humans having a slightly higher median, and a wide range of readability scores, suggesting no marked difference in text clarity between them. Outliers indicate texts with unusual word variety or simplicity. These findings suggest that lexical diversity and readability alone may not clearly differentiate bots from humans, pointing to the potential need for additional linguistic features to distinguish between them effectively. The data indicates that bots may be mimicking human language patterns closely.

Back to Top

Topic Modeling

Back to Top

Sentiment Over Time

The line graph shows monthly tweet sentiment counts over three years, revealing a consistent prevalence of positive sentiment tweets and a steady, lower frequency of negative sentiment tweets. Neutral tweets fluctuate moderately without showing a clear long-term trend. While no distinct seasonal patterns are immediately visible, a deeper analysis might reveal such trends influenced by external events or cultural shifts. The graph's reliability depends on consistent data collection and sentiment analysis methods across the observed period. Overall, the data suggests a generally positive expression trend on social media.

Back to Top

Sentiment and Engagement Correlation

The scatter plot examines the interplay between sentiment scores and retweet counts on a social media platform, differentiating between bot (1) and human (0) accounts. Sentiment scores span from highly negative to highly positive for both account types, with a notable concentration around the neutral midpoint. Most tweets, from bots and humans alike, garner only a handful of retweets, evidenced by the clustering of data points at the lower end of the retweet count.

Both human and bot accounts display a comparable range of sentiment scores and retweet counts, with no distinct pattern emerging that associates sentiment with the likelihood of being retweeted. This observation suggests sentiment score is not a definitive indicator of a tweet's retweet potential.

The similarity in the distribution of sentiment scores and retweet counts for both bots and humans may indicate that bots are emulating human engagement patterns. However, the dense overlap of data points poses challenges in distinguishing between the two, highlighting the need for more intricate analysis to discern underlying trends.

In summary, the scatter plot reveals the intricate nature of social media engagement, where neither sentiment nor retweets clearly differentiate human from bot activity, calling for more advanced methods to parse these complex interactions.

Back to Top

Machine Learning Models

Random Forest Classification

Back to Top

Logistic Regression

Back to Top

Support Vector Machines (SVM)

Back to Top

K-Nearest Neighbors (KNN)

Back to Top

Gradient Boosting Machines (GBM), including XGBoost, LightGBM, and CatBoost

Back to Top

Majority Voting: Joined ALL three ensemble methods above

Back to Top

Neural Networks/Deep Learning

Multilayer Perceptrons (MLP)

MLPs are powerful because they can learn complex patterns by adjusting connections in a network of layers. Our results show the model is quite good at recognizing non-bots but struggles to accurately identify bots, as seen in the high recall for non-bots and low recall for bots. The precision being equal for both classes means our model is consistent in its accuracy rate across both bot and non-bot predictions. Overall, the model's accuracy is around 49%, indicating it's not performing well in distinguishing between bots and non-bots, similar to making a guess. The challenge now is to improve the model's ability to detect bots without losing its performance on non-bots, possibly by further tuning the model or using more sophisticated neural network architectures. So I'm going to work with RNN, BERT and LSTM

Back to Top

Recurrent Neural Networks (RNN)

Back to Top

Long Short-Term Memory (LSTM) Networks

Back to Top

Prediction for RNN and LSTM

Back to Top

Enhancing models with feature selections

Improving Features

Back to Top

Random Forest Classification Models

Feature Selection 1 using numerical variables only.

Back to Top

Feature Selection 2 using numerical variables only

Back to Top

Feature Selection 3 using categorical variables only.

Back to Top

Feature Selection 4 using categorical variables excluding some features

Overall, enhancing features selections aren't showing any improvements. So, I will stick with neural nets above that has already been improved

Back to Top

Using LIME (explainable AI)

Back to Top

Lime topic Analysis

Human Topics:

Personal Action and Creativity: guess send know build suddenly usually continue try reflect music

Formal and Informative Content: article example member mr certain market chair according magazine road

Recreational and Social Activities: dog social head sport thousand event list card operation nearly

Achievement and Challenge: end pressure true star edge college perform prevent court general

Duty and Aspiration: soldier final structure support language development professional action push dream

Bot Topics:

Abstract Experiences and Aspirations: experience reveal future painting guess agent best person smile behavior

Interpersonal and Normative Discussions: argue remain example son certain sister quite make rule section

Strategic and Analytical Focus: ground military cost event total region million use apply hand

Social Connectivity and Lifestyle: pressure general right meet lead popular doctor store couple sit

Professional Development and Gratitude: development young structure professional cold spend challenge dream official thank

Distinguishing Factors:

Human topics tend to focus on personal actions, creativity, and specific social or recreational activities, suggesting content that might be more experiential or narrative in nature.

Bot topics, on the other hand, appear to cover a broader range of abstract experiences, strategic discussions, and general advice, potentially indicating content that is more informational, advisory, or aimed at broader audience engagement.

Back to Top

SHAP (explainable AI)

SHAP plot visually explains the contribution of each feature to a specific model prediction. Features pushing the prediction higher (towards being a bot) are shown in red, and those pushing it lower (towards being human) are in blue. The base value is the model's average output over the dataset, and f(x) is the model's output for this specific instance. If f(x) is higher than the base value, the model leans towards predicting 'bot'; if lower, 'human'. Here, f(x) is very close to the base, suggesting a borderline decision, but slightly more towards 'bot'.

Back to Top